System and method for data transfer from jdbc to a data warehouse layer in a massively parallel or distributed database environment

ABSTRACT

A system and method is described for database split generation in a massively parallel or distributed database environment including a plurality of databases and a data warehouse layer providing data summarization and querying functionality. A database table accessor of the system obtains, from an associated client application, a query for data in a table of the data warehouse layer, wherein the query includes a user preference. The system obtains table data representative of properties of the table, and determines a splits generator in accordance with one or more of the user preference or the properties of the table. The system generates, by the selected splits generator, table splits dividing the user query into a plurality of query splits, and outputs the plurality of query splits to an associated plurality of mappers for execution by the associated plurality of mappers of each of the plurality of query splits against the table.

CLAIM OF PRIORITY

This application claims the benefit of priority to U.S. Provisionalpatent application titled “SYSTEM AND METHOD FOR HIGH SPEED DATATRANSFER FROM JDBC TO A DATA WAREHOUSE LAYER”, Application No.62/056,424, filed Sep. 26, 2014, which is herein incorporated byreference.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD OF INVENTION

The embodiments described herein are generally related to dataprocessing, and are particularly related to systems and methods fordatabase split generation in massively parallel or other distributeddatabase environments.

BACKGROUND

Data processing frameworks are available for distributed storage andprocessing of large sets of data on commodity hardware. One such dataprocessing framework is Apache Hadoop™ which is an open source frameworkfor distributed storage and processing of large sets of data oncommodity hardware. The Hadoop framework can be used to process largeamounts of data in a massively parallel or other distributed mannerthereby enabling businesses to quickly gain insight from massive amountsof both structured as well as unstructured data.

The Hadoop framework is enterprise ready in part because it provides forthe storage and processing of vast quantities of data in a storage layerthat scales linearly. In this regard, Hadoop Distributed File System(HDFS) is a technology providing for efficient scale out of a storagelayer. Environments such as HDFS provide a fault-tolerant environmentwhich is designed to be deployed within a distributed computinginfrastructure, using relatively low-cost commodity hardware. Such anenvironment provides high throughput access to application data, and isparticularly suitable for applications that have very large data sets(e.g., machine learning).

Many specialized engines are available for enabling interaction with thedata in a wide variety of ways including batch access, real-time access,and combinations of batch and real-time access. Apache Hive™ is the mostwidely adopted technology for accessing massive amounts of data such asmight be organized and stored in Hadoop and is, essentially, a datawarehouse having tables similar to tables in a relational database.Engines such as Hive enable easy data summarization andad-hoc/interactive queries via a structured query language (SQL) likeinterface for large datasets (e.g., petabytes of data) stored in HDFS.

Table and storage management interface layers such as the HCataloginterface enable users with different data processing tools to moreeasily read and write data relative to the Hadoop environment. HCataloguses Hive's command line interface for issuing commands to define dataand to explore metadata.

Latency for Hive queries, however, is generally very high even forrelatively small data sets owing in part to the batch processing ofHadoop jobs which can at times incur substantial overheads in jobsubmission and scheduling. In addition, tools such as HIVE that use adatabase abstraction layer such as HCatalog can divide a query intomultiple pieces and execute them separately against a database. However,these queries are not executed atomically but instead are executed atdifferent points in time. As such, the results of each query whencombined could violate the read-consistent rule relative to databaseretrieval protocol rules.

SUMMARY

In accordance with an embodiment, described herein is a system andmethod for database split generation in a massively parallel or otherdistributed database environment including a plurality of databases anda data warehouse layer providing data summarization and querying of theplurality of databases. In an example embodiment, the system includesone or more microprocessors and a database table accessor running on theone or more microprocessors. The database table accessor operates toobtain, from an associated client application, a query for data in atable of the data warehouse layer, wherein the query comprises querydata representative of a user query and user preference datarepresentative of a user preference. The database table accessor furtheroperates to obtain table data representative of one or more propertiesof the table, and to determine a splits generator in accordance with oneor more of the user preference or the one or more properties of thetable. The database table accessor operates to generate, by the selectedsplits generator, table splits dividing the user query into a pluralityof query splits, and to output the plurality of query splits to anassociated plurality of mappers for execution by the associatedplurality of mappers of each of the plurality of query splits againstthe table. The table splits may be one or more of partition-basedsplits, size-based splits, ROWID-based splits or others as may benecessary or desired.

In accordance with a further embodiment, described herein is a systemand method for database split generation and consistent reads in anassociated massively parallel or other distributed database environmentincluding a plurality of databases and a data warehouse layer providingdata summarization and querying of the plurality of databases. In anexample embodiment, the system includes one or more microprocessors anda database table accessor running on the one or more microprocessors.The database table accessor operates to obtain, from an associatedclient application, a query for data in a table of the data warehouselayer, wherein the query comprises query data representative of a userquery and user preference data representative of a user preference. Thedatabase table accessor further operates to obtain table datarepresentative of one or more properties of the table, and to determinea splits generator in accordance with one or more of the user preferenceor the one or more properties of the table. The database table accessorfurther operates to obtain, from the data warehouse layer, recordcurrent system change number (SCN) data, the SCN data beingrepresentative of a logical internal time stamp used by the plurality ofdatabases of the associated massively parallel or other distributeddatabase environment. The database table accessor operates to generate,by the selected splits generator, table splits dividing the user queryinto a plurality of query splits, to associate the SCN data with each ofthe plurality of query splits, and to output the plurality of querysplits together with the SCN data to a plurality of associated mappersfor execution by the plurality of mappers against the table as querytasks using the SCN data to provide consistent reads between the querytasks.

In accordance with a further embodiment, described herein is a systemand method for high-speed data transfer from JDBC to a data warehouselayer as will be shown and described below.

In accordance with a further embodiment, described herein is a systemand method for query processing with table-level predicate pushdown aswill be shown and described below.

In accordance with a further embodiment, described herein is a systemand method for efficient connection management in a distributed databaseenvironment as will be shown and described below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system for dynamically generating input splitsbased on table properties and/or user preferences for use in a massivelyparallel or other distributed environment, in accordance with anembodiment.

FIG. 2 is a flowchart that illustrates a method for dynamicallygenerating input splits based on table properties and/or userpreferences for use in a massively parallel or other distributedenvironment, in accordance with an embodiment.

FIG. 3 is a flowchart that illustrates a method for dynamicallygenerating input splits based on table properties and/or userpreferences for use in a massively parallel or other distributedenvironment, in accordance with a further embodiment.

FIG. 4 illustrates a system for providing consistent reads between tasksusing efficient splits in a parallel data processing system, inaccordance with an embodiment.

FIG. 5 is a flowchart that illustrates a method for providing consistentreads between tasks using efficient splits in a parallel data processingsystem, in accordance with an embodiment.

FIG. 6 is a functional control flowchart that illustrates a method fordynamically generating input splits based on table properties for use ina massively parallel or other distributed environment, in accordancewith an embodiment.

FIG. 7 illustrates a system for dynamically generating partition-basedsplits based on table properties and/or user preferences for use in amassively parallel or other distributed environment, in accordance withan embodiment.

FIG. 8 is a flowchart that illustrates a method for dynamicallygenerating partition-based splits based on table properties and/or userpreferences for use in a massively parallel or other distributedenvironment, in accordance with an embodiment.

FIG. 9 illustrates a system for dynamically generating size-based splitsbased on table properties and/or user preferences for use in a massivelyparallel or other distributed environment, in accordance with anembodiment.

FIG. 10 is a flowchart that illustrates a method for dynamicallygenerating size-based splits based on table properties and/or userpreferences for use in a massively parallel or other distributedenvironment, in accordance with an embodiment.

FIG. 11 illustrates a system for dynamically generating ROWID-basedsplits based on table properties and/or user preferences for use in amassively parallel or other distributed environment, in accordance withan embodiment.

FIG. 12 is a flowchart that illustrates a method for dynamicallygenerating ROWID-based splits based on table properties and/or userpreferences for use in a massively parallel or other distributedenvironment, in accordance with an embodiment.

FIG. 13 illustrates a system for providing rapid transfer of data from adatabase driver to generic writable interfaces in a data warehouselayer, in accordance with an embodiment.

FIG. 14 is a flowchart that illustrates a method for providing rapidtransfer of data from a database driver to generic writable interfacesin a data warehouse layer, in accordance with an embodiment.

FIG. 15 illustrates a system for providing query processing withtable-level predicate pushdown in a data warehouse layer, in accordancewith an embodiment.

FIG. 16 is a flowchart that illustrates a method for providing queryprocessing with table-level predicate pushdown in a data warehouselayer, in accordance with an embodiment.

FIG. 17 illustrates a system for providing efficient connectionmanagement in a distributed database environment, in accordance with anembodiment.

FIG. 18 is a flowchart that illustrates a method for providing efficientconnection management in a distributed database environment, inaccordance with an embodiment.

DETAILED DESCRIPTION

In accordance with an embodiment, described herein is a system andmethod for database split generation in a massively parallel or otherdistributed database environment including a plurality of databases anda data warehouse layer providing data summarization and querying of theplurality of databases.

In accordance with an embodiment, a data processing infrastructure, suchas Hadoop, can include components that are central to effectivelydealing with petabyte-scale problems: a distributed file system toprovide fault-tolerant storage while scaling horizontally, and aparallel processing framework that can break processing jobs into tasksand distribute the tasks across a data processing cluster, or amassively parallel or other distributed environment.

In accordance with an embodiment, a data processing job can includeseveral types of tasks including: a map task that filters and interpretsinput data to produce key-value pairs; and a reduce task that summarizesand aggregates the map results to produce final outputs.

To better define data processing jobs in a data processing framework, aplurality of application program interfaces (APIs) have been developedto allow users and developers to express their data processing jobs inthe language they deem most suitable. As an illustrative example, Hiveis a data warehouse system that provides such an API, wherein the APIcan be used to facilitate querying and managing large datasets using anSQL-like language called HiveQL™, wherein the datasets can either residein a Hadoop distributed file system or any other equivalent differentsystem.

In accordance with an embodiment, when data to be queried is stored in arelationship table, such as an Oracle™ database table, and particularlywhen the table is large, a query is broken into multiple query splitsfor use by the massively parallel or other distributed environment.

In accordance with an embodiment, described herein is a system andmethod for database split generation in a massively parallel or otherdistributed database environment including a plurality of databases anda data warehouse layer providing data summarization and querying of theplurality of databases.

Dynamic Database Split Generation

In accordance with an embodiment, described herein is a system andmethod for strategically and dynamically generating input splits basedon table properties and user preferences for use in a massively parallelor other distributed environment including a plurality of databases anda data warehouse layer providing data summarization and querying of theplurality of databases.

In accordance with an embodiment, the system includes a database tableaccessor and a data warehouse layer, where the database table accessor,upon receiving a query for data in a table, can select a splitsgenerator based on properties of the table and user preferences, togenerate table splits and split the query into one or more query splitsin accordance with the table splits. Each query split can then beexecuted against an external table created in the data warehouse layerto retrieve data from the table for processing.

In accordance with an embodiment, the database table accessor caninclude a splits generators component with an enumeration of splitterkinds indicating user hints, each splitter kind associated with acorresponding splits generator. In accordance with an embodiment, givena name of a splitter kind, the splits generators component can return anappropriate splits generator.

Additionally, the database table accessor can include an input formatcomponent responsible for generating query splits itself or invoking asplits generator to generate the query splits.

In accordance with an embodiment, the external table can be created at auser interface and managed by the database table accessor. Additionally,the external table can include a user-defined splitter kind as aproperty. In accordance with an embodiment, metadata of the externaltable can be stored in the data warehouse layer and used to access datain the database table.

In accordance with an embodiment, when creating table splits of a table,the table is logically split so that each split of the table can beaccessed from a query level via an external tool such as, for example,HIVE or any other similar or equivalent technology tools for accessingmassive amounts of data such as might be organized and stored in Hadoopenabling easy data summarization and ad-hoc/interactive queries via aSQL-like interface for large datasets (e.g., petabytes of data) storedin HDFS.

In accordance with an embodiment, a table can be split based on varioussplitter kinds, which indicate a user's preferences how the table shouldbe split. In accordance with an embodiment, the splitter kinds caninclude splitting a table based on partitions, sizes, row limits andROWID ranges. The database table accessor, when splitting a table,additionally takes considerations of properties of the table, and canstrategically and dynamically split the table based on these properties

As an illustrative example, the database table accessor may split apartitioned table differently, even with a user hint that the table besplit based on a ROWID range. In accordance with an embodiment,splitting a partitioned table by a ROWID range could generate duplicaterows in two different splits, causing problems in the reduce/aggregationphase in a map reduce job.

In accordance with an embodiment, the database table accessor can use astrategy or heuristic to split a table based on a plurality of factors,including a user hint, the size of the table, whether the table ispartitioned, whether splitting the table in a particular way can improvethe overall performance of the system. For example, if a table is small,the database table accessor can take the whole table as a single splitand provide the split to one map reduce job for processing.

In accordance with an embodiment, the above-described approach providesan architecture for dynamically generating query splits based on tableproperties and user hints, and for efficiently creating new splitgenerators. In addition, the system is highly configurable withadditional user preferences and associated splits generators toaccommodate various requirements of users or client applications.

FIG. 1 illustrates a system for dynamically and strategically splittinga table based on one or more of properties of the table or user hintsfor a massively parallel or other distributed environment, in accordancewith an embodiment.

As shown in FIG. 1, by way of example, the system can include anapplication server or database access environment 100; a physicalcomputer processor/CPU and storage 101; a database table accessor (e.g.,a database table accessor for Hadoop) 110; and a data warehouse layer(e.g., a HIVE component) 108; wherein the database access environment100 can receive from a client application 122 a query 124 (e.g., a SQLquery) for data in a table 105 in a database 104.

In accordance with an embodiment, the data requested from the table 105can be passed 171 to a data processing cluster (e.g., a Hadoop cluster)172 for processing by map reduce jobs, e.g., 174, distributed across thecluster. The aggregation outputs can then be returned 173 to the systemfor presentation to the client application 122.

As further shown in FIG. 1, the database table accessor 110 includes, inaccordance with an embodiment herein, a splits generators component 112and an input format component 118. In accordance with an embodiment, thesplits generators component 112 includes a splitter kind enumeratorobject 114 that wraps a plurality of splitter kinds and a plurality ofinput splits generators 116, each of which is associated with aparticular splitter kind. In accordance with an embodiment, a splitterkind is a user input specifying how a table should be split. Inaccordance with an embodiment, given the name of a splitter kind, thesplits generators component can return a splits generator associatedwith the splitter kind. In accordance with another embodiment, givenproperties of the table object of the search query, the splitsgenerators component can return a splits generator associated with asplitter kind selected in accordance with the table properties.

Listing 1, as shown below, illustrates an implementation of the splitsgenerators component in accordance with an embodiment.

Listing 1 public enum SplitsGeneratorKind { /**  * Generates one Split. */ SIMPLE(new OracleSimpleSplitsGenerator( )) privateOracleSplitsGenerator generator;SplitsGeneratorKind(OracleSplitsGenerator value) { generator = value; }/**  * Obtain an InputSplit generator for a given splitter kind.  *  *@return an instance of OracleSplitsGenerator  */ OracleSplitsGeneratorgetGenerator( ) { return generator; }

As shown in Listing 1 above, in accordance with an embodiment, thesplits generators component can return a simple splits generator forgenerating a single split, when the table includes a smaller number ofrows; or an appropriate splits generator for a given splitter kind.

As further shown in FIG. 1, the input format component can beresponsible for generating table splits based on properties of the tableand/or user preferences, and generating query splits in accordance withthe generated table splits. The input format component can generatequery splits itself, or use 120 a splits generator 121 selected by asplits generator selection logic 129 to generate the query splits.

In accordance with an embodiment, the splits generator selection logic129 can select a splits generator using a pre-defined strategy incombination with properties of a table and/or user preferences retrieved151 from the data warehouse layer 108. In accordance with an embodiment,once a selection is made, the input format component can obtain 117 theselected splits generator from the splits generators component 112 usingthe corresponding splitter kind.

In accordance with embodiment, an example implementation of the inputformat component can be illustrated using pseudocode as shown in Listing2:

Listing 2 fetchTableProperties( ); if TABLE_PARTITIONED then UsePartionedSplitsGenerator; else if FULL_SELECT AND SPLIT_HINT =ROWID_RANGE then Use RowidRangeSplitsGenerator; endif InputSplits splits= splitsGenerator.getGenerator( ).getInputSplits(...);

As shown in Listing 2, when the input format component is invoked,properties of a database table are fetched and examined, and internallogic is used to select an appropriate splits generator based on userpreferences (user “hints”) and/or table properties. For example, if thedatabase table is partitioned, the database table accessor 110 canselect a partition-based splits generator; if a query is received fordata in the table based on a user-defined size enabling the table to besplit into multiple ranges for optimal processing, the database tableaccessor 110 can select a size-based splits generator; and if a queryreceived is for selecting all rows in a table, and a user preference orhint received is to split the table by a ROWID range, the database tableaccessor 110 can select a ROWID range based splits generator. Inaccordance with an embodiment, a plurality of other types of splitsgenerators can be chosen based on the table properties and the userpreferences or hints and/or the table properties as may be necessary ordesired. Once an appropriate splits generator is selected, it can beused to generate splits of the table, and to split the query into aplurality of query splits in accordance with the generated table splits.

In an accordance with an embodiment, each splits generator can implementa common interface, e.g., OracleSplitsGenerator, and can include amethod, e.g., getInputSplits( ), which returns the InputSplits generatedby the splits generator. Similarly, each query split can implement acommon interface, e.g., OracleHiveSplit, which can include agetSqlStringToAppend( ) method and a getBinds( ) method, wherein theformer method returns a split specific SQL query, i.e. a InputSplits,and the latter method returns binds specific to a InputSplits.

In accordance with an embodiment, the data warehouse layer 108, e.g.,HIVE, can be an abstraction layer over the data processing cluster, andenable accessing the database 104 using SQL queries while keepingtransaction models in a transaction. As shown in FIG. 1, the datawarehouse layer 108 includes a metadata store 150 containing dataincluding obtaining table data representative of one or more propertiesof the table 146 wherein a splits generator is determined in accordancewith one or more of the user preference or the one or more properties ofthe table, where a user can execute commands or queries to interact withthe database 104 via execution of the query splits resulting from thesplits generator as determined.

In accordance with an embodiment, an external table 146 is selectivelycreated in the data warehouse layer 108 using the database tableaccessor 110, where definitions of the external table stored in themetadata store 150 can be used to access data in the database table 105.In accordance with an embodiment, data of the database table 105 wouldnot be deleted if the external table is dropped. In accordance with anembodiment, the columns of the external table 146 can be the same withthose of the database table 105, but the types of a particular columncan be different in the two tables 146, 105.

In accordance with an embodiment, a query can be executed against theexternal table 146 to retrieve data from the database table, i.e. 105.In accordance with an embodiment, the data warehouse layer 108 canaccess the database 104 via a data source abstraction component 106,e.g., HCatalog. The data source abstraction component 106 can presentusers with a relational view of data from various types of data sources(e.g., database or files) and ensure that users need not be concernedwith where or in what format the data is stored.

In accordance with an embodiment, the database table accessor 110 canuse a data definition language (DDL), e.g., HIVE DDL, to managedefinitions in both the metadata store 150 and the database metadatasimultaneously and consistently. In accordance with an embodiment, thedatabase table accessor 110 can associate itself with the external table146 using e.g., a STORED BY clause when the external table is created.

As an illustrative example, an external table can be created in the datawarehouse layer 108 using a DDL syntax shown in Listing 3:

Listing 3 CREATE EXTERNAL TABLE sales_test ( prod_id int, cust_nameSTRING, TIME_ID timestamp ) STORED BY‘oracle.hcat.osh.OracleStorageHandler’ TBLPROPERTIES ( ‘oracle.jdbc.url’= ‘jdbc:oracle:thin:@localhost:1521:orcl’, ‘oracle.jdbc.username’ =‘scott’, ‘oracle.jdbc.password’ = ‘tiger’, ‘oracle.hcat.osh.tableName’ =‘sales_test’, ‘oracle.hcat.osh.splitterKind’ = ‘ROWID_RANGE’ )

In accordance with an embodiment, in Listing 3, an external table“sales_test” is created with column names “prod_id”, “cust_name” and“TIME_ID” and managed by a database table accessor“oracle.hcat.osh.OracleStorageHandler”. Additionally, the external tableincludes a plurality of properties, such as the name of the databasetable and a user hint that the database table should be split by a ROWIDrange.

In accordance with an embodiment, the metadata of the external table isstored in the metadata store 150 in the data warehouse layer 108. When aselect query is executed against the external table 146, the tablemetadata is used to connect to the database 104 to fetch data from atable based on selected criteria such as, for example, a ROWID_Range,and present the data in the data warehouse layer 108.

As further shown in FIG. 1, in accordance with an embodiment,information about each of the query splits generated by the input formatcomponent can be embedded into a separate input split object, e.g., 128,130 or 140. Each input split object can spawn a separate a recordreader, e.g., 152, 154 or 160, which executes the query split againstthe external table 146 to retrieve records from the database table 105.In accordance with an embodiment, each record reader can further spawn aseparate mapper, e.g., 162, 164 or 170, and correspondingserializers/de-serializers 153, 155 or 161 to process the recordsretrieved by each of the plurality of query splits as they are executedindividually. In accordance with an embodiment, each mapper 162, 164 or170 can be part of a map-reduce jobs in the data processing cluster.

FIG. 2 is a flowchart that illustrates a method 200 for dynamicallygenerating input splits based on one or more of table properties or userpreferences for use in a massively parallel or other distributedenvironment, in accordance with an embodiment.

As shown in FIG. 2 and with continued reference also to FIG. 1, inaccordance with an embodiment, at step 211, one or more computers 101(FIG. 1) including a database table accessor 110 and a data warehouselayer 108 for accessing a database 104 are provided.

At step 213, an external table 146 (FIG. 1) is created in the datawarehouse layer 108 for a table 105 in a database 104, wherein theexternal table 146 is managed by the database table accessor 110 andincludes one or more of user preferences or table properties on how thedatabase table 105 should be split.

At step 215, the database table accessor 110 receives a query 124 thatselects all rows in the HIVE/HCatalog table, and obtains properties ofthe table and the user preference.

At step 217, the database table accessor 110 selects a splits generatorbased on the table properties and the user preference.

At step 219, the database table accessor 110 uses the selected splitsgenerator 121 to split the query into multiple query splits 128, 130, or140.

At step 221, each query split is provided to a record reader 152, 154,or 160 that executes that query split against the external table 146.

At step 223, records from the database table 105 retrieved by each querysplit are provided to a mapper 162, 164, or 170 for processing by a dataprocessing cluster.

FIG. 3 is a flowchart that illustrates a method 300 in accordance withan example embodiment for dynamic database split generation in amassively parallel or other distributed database environment including aplurality of databases and a data warehouse layer providing datasummarization and querying of the plurality of databases. As shown inFIG. 3 and with continued reference also to FIG. 1, in accordance withan embodiment, at step 310, the database table accessor 110 obtains,from an associated client application 122, a query 124 for data in atable of the data warehouse layer. In the example embodiment, the query124 comprises query data representative of a user query and userpreference data representative of a user preference.

At step 320, the database table accessor 110 obtains table datarepresentative of one or more properties of the table and, at step 330the database table accessor 110 determines a splits generator inaccordance with one or more of the user preference or the one or moreproperties of the table.

At step 340 the database table accessor 110 generates, by the selectedsplits generator, table splits dividing the user query into a pluralityof query splits.

At step 350 the database table accessor 110 outputs the plurality ofquery splits to an associated plurality of mappers 162, 164, or 170 forexecution by the associated plurality of mappers of each of theplurality of query splits against the database table 105.

Consistent Reads Between Tasks

As noted above, overall, tools such as HIVE that use a databaseabstraction layer such as HCatalog can divide a query into multiplepieces and execute them separately against a database. However, thesequeries are not executed atomically but instead are executed atdifferent points in time. As such, the results of each query whencombined could violate the read-consistent rule relative to databaseretrieval protocol rules.

Accordingly, in accordance with a further embodiment, the system andmethod for dynamic database split generation in a massively parallel orother distributed database environment as described above also providesconsistent reads between tasks using the efficient splits in a paralleldata processing cluster. In accordance with an embodiment the databasetable accessor 110, before generating splits of the table dividing theuser query into the plurality of query splits, can record the currentsystem change number (SCN) in the database during or at the time of thesplits generation, save the SCN as part of each generated query split,and during execution of the query splits using different tasks, use theSCN from each query split to achieve consistent reads between the tasks.

In accordance with an embodiment, an SCN is a database orderingprimitive and represents a logical, internal time stamp used by adatabase, e.g., Oracle Database. SCNs order events that occur within thedatabase, which is necessary to satisfy the atomicity, consistency,isolation, and durability (ACID) properties of a transaction thatguarantee that database transactions are processed reliably. SCNs occurin a monotonically increasing sequence. An observed SCN indicates alogical point in time. If one event has a lower SCN than another event,then it occurred at an earlier time with respect to the database.Several events may share the same SCN, which means that they occurred atthe same time with respect to the database. As such, an SCN can be usedto exclude records added after the SCN was created if necessary ordesired.

In accordance with an embodiment, the saved SCN can be checked forvalidity before it is set on a SQL statement for execution. The databasetable accessor 110 can check whether a generated split includes a SCN,or whether the saved SCN is in a valid format, or an object includingthe SCN is properly initialized. If the validity check fails on one ormore query splits of the plurality of query splits, an error isindicated, and every query split is than executed anew.

FIG. 4 illustrates a system for providing consistent reads between tasksusing efficient splits in a parallel data processing cluster, inaccordance with an embodiment.

As illustrated in FIG. 4, before the input format component generatesinput splits, it can obtain 451 the current SCN from the database tableand record the current SCN in the database table accessor 110. Duringinput splits generation, the input format component can save the currentSCN (e.g., SCN) 428, 430 and 440 as part of each generated query split128, 130 or 140. During execution of the query splits, the SCN can beset on a SQL statement and be used 461 to achieve consistent readsacross the different tasks.

The following pseudocode as shown in Listing 4 below illustrates anexample implementation of the method for providing consistent readsbetween tasks using efficient splits in a parallel data processingcluster, in accordance with an embodiment.

Listing 4 // before generation of splits, record the SCN long scn =oracleConnection:getCurrentSCN ( ); // during generation of splits, saveSCN as a part of Split InputSplit splits = new OracleHiveSplit(scn); //during execution, use saved SCN from split during execution long scn =oracleHiveSplit:getScn( ); IF scn NOT VALID THEN THROW EXCEPTION ELSESET SCN ON STATEMENT ENDIF

FIG. 5 is a flowchart that illustrates a method 500 in accordance withan example embodiment for dynamic database split generation in amassively parallel or other distributed database environment providingconsistent reads between the plurality of query splits. As shown in FIG.5 and with continued reference also to FIG. 1, in accordance with anembodiment, at step 510, the database table accessor 110 obtains, froman associated client application 122, a query 124 for data in a table ofthe data warehouse layer. In the example embodiment, the query 124comprises query data representative of a user query and user preferencedata representative of a user preference.

At step 520, the database table accessor 110 obtains table datarepresentative of one or more properties of the table and, at step 530the database table accessor 110 determines a splits generator inaccordance with one or more of the user preference or the one or moreproperties of the table.

At step 535 the current system change number is obtained.

At step 540 the database table accessor 110 generates, by the selectedsplits generator, table splits dividing the user query into a pluralityof query splits.

At step 550 the database table accessor 110 outputs the plurality ofquery splits to the associated plurality of mappers 162, 164, or 170 forexecution by the associated plurality of mappers of each of theplurality of query splits against the database table 105. In theembodiment each of the plurality of query splits sent to the associatedplurality of mappers 162, 164, or 170 includes data representative ofthe current SCN.

FIG. 6 is a functional control flowchart that illustrates a method fordynamically generating input splits based on table properties for use ina massively parallel or other distributed environment, in accordancewith an embodiment. With reference now to that Figure, the databasetable accessor 110 initiates a control function at control block 610 todetermine a splitter kind using the split generator selection logic 129(FIG. 1). By the split generator selection logic 129, the database tableaccessor 110 can determine from the metadata store 150 whether the datain the table is partitioned at control block 620 or whether the data inthe table is unpartitioned at control block 622.

When it is determined at control block 620 that the data in the table ispartitioned, the split generator selection logic 129 selects apartition-based splits protocol at control block 650 for generating apartition-based splits generator 121 (FIG. 1). Partition-based splitswill be described below in detail with reference in particular to FIGS.7 and 8.

With continued reference to FIG. 6, however, in accordance with anembodiment, when it is determined at control block 622 that the data inthe table is unpartitioned, the database table accessor 110 initiates acontrol function at control block 630 to determine whether the data inthe table is of a fixed size, and a control function at control block640 when the table is not of a fixed size to determine a number ofrecords in the table. When it is determined at control blocks 622, 632that the data in the table is unpartitioned and not of a fixed size, thesplit generator selection logic 129 selects a limit-based splitsprotocol at control block 656 for generating a limit-based splitsgenerator 121 (FIG. 1).

However, when it is determined at control blocks 622, 630 that the datain the table is unpartitioned and of a fixed size, the database tableaccessor 110 can determine from the metadata store 150 whether the datain the table is of an unknown database chunk size at control block 640or whether the data in the table is of a known size at control block642. When it is determined at control block 640 that the data in thetable is of an unknown database chunk size, the split generatorselection logic 129 selects a size-based splits protocol at controlblock 652 for generating a size-based splits generator 121 (FIG. 1).Size-based splits will be described below in detail with reference inparticular to FIGS. 9 and 10. Similarly, when it is determined atcontrol block 642 that the data in the table is of an known databasechunk size, the split generator selection logic 129 selects aROWID-based splits protocol at control block 654 for generating aROWID-based splits generator 121 (FIG. 1). ROWID-based splits will bedescribed below in detail with reference in particular to FIGS. 11 and12.

Partition-Based Splits

In accordance with a further embodiment, a method for databasepartition-based split generation in an associated massively parallel orother distributed database environment including a plurality ofdatabases and a data warehouse layer providing data summarization andquerying of the plurality of databases is provided. A query for data ina table of the data warehouse layer is obtained from an associatedclient application, the query comprising query data representative of auser query and user preference data representative of a user preference.The method further includes obtaining, from the data warehouse layer,table data representative of one or more properties of the table,wherein the table data comprises partition data representative of apartition scheme of the table as having a partitioned topology whereinthe table is logically divided into one or more partitions or anun-partitioned topology wherein the table is logically undivided. Themethod further includes selecting a partition-based splits generator inaccordance with one or more of the query data indicating a userpreference for a partition-based splits generator or the partition dataindicating the partition scheme of the table as having the partitionedtopology, and generating, by the selected partition-based splitsgenerator, one or more table splits for each of the one or morepartitions of the table, the one or more table splits dividing the userquery into a plurality of query splits. The method further includesoutputting the plurality of query splits to a plurality of associatedmappers for execution by the plurality of associated mappers of each ofthe plurality of query tasks against the table.

FIG. 7 illustrates a system for generating query splits based onpartition and sub-partitions of a database table for use in a massivelyparallel or other distributed environment. Overall and in accordancewith an embodiment, partitioning allows a table to be subdivided intosmaller pieces, where each piece of such a database table is called apartition. Each partition has its own name, and may optionally have itsown storage characteristics. It is to be appreciated that partitioningenhances the performance, manageability, and availability of a widevariety of applications and helps reduce the total cost of ownership forstoring large amounts of data. From the perspective of a databaseadministrator, a partitioned table has multiple pieces that can bemanaged either collectively or individually. This gives theadministrator considerable flexibility in managing partitioned tables.However, from the perspective of the application, a partitioned table isidentical to a non-partitioned table; no modifications are necessarywhen accessing a partitioned table using SQL queries and DML statements.

Therefore, in accordance with an embodiment, described herein is asystem and method for generating query splits based on partition andsub-partitions of a database table for use in a massively parallel orother distributed environment.

In accordance with an embodiment, a user can first create an externaltable in a data warehouse layer using a partitioned splitter as thesplitter kind, wherein the external table is managed by a database tableaccessor and can be used to access a partitioned table in a database.When a query for selecting rows in the table is received, the databasetable accessor operates to choose an appropriate splits generator,execute an explain plan to derive a list of valid partitions relevant tothe query, calculate the number of bytes and data included for eachvalid partition.

In accordance with an embodiment, the database table accessor cancombine or split partitions/sub-partitions in the list of validpartitions based on a maximum split size defined in the massivelyparallel or other distributed environment (data processing cluster) suchas Hadoop. If the size of a partition exceeds the maximum split size,the partition can be divided into multiple splits; if the total size ofmultiple partitions is less than the defined maximum size, the multiplepartitions can be combined onto one split.

In accordance with an embodiment, an external table that includes apartition-based splitter kind is created as illustrated in the exampleof Listing 5 set out below:

Listing 5 CREATE EXTERNAL TABLE sales_test ( prod_id int, cust_nameSTRING, TIME_ID timestamp ) STORED BY′oracle.hcat.osh.OracleStorageHandler′ TBLPROPERTIES ( ′oracle.jdbc.url′= ′jdbc:oracle:thin:@localhost:1521:orcl′, ′oracle.jdbc.username′ =′scott′, ′oracle.jdbc.password′ = ′tiger′, ′oracle.hcat.osh.tableName′ =′sales_test′, ′oracle.hcat.osh.splitterKind′ = ‘PARTITIONED_SPLITTER’ );

In accordance with an embodiment, when a query is received, the databasetable accessor can select a partition-based splits generator 721 basedon the specified partition-bases splitter kind. The partitioned basedsplits generator can execute an explain plan 722 to inquire about thetopology of the table and retrieve 751 the table partition information.In accordance with an embodiment, the explain plan is a structure querylanguage (SQL) statement that displays how a database executes SQLstatements, and can be used to view how the database accessespartitioned objects for specific queries.

In accordance with an embodiment, the partition-based splits generatorcan use the explain plan to obtain partition number ranges and generatepartition names from the partition number ranges for the given query. Inparticular, the database table accessor can analyze the partitioninformation with respect to the query to derive a valid list ofpartitions. As an illustrative example, if the query includes apredicate that narrows the selection to the first quarter and thirdquarter of the sales table, the valid list of partitions can includepartitions for these two quarters; the partitions for the second andfourth quarters would be unwanted partitions and pruned.

In accordance with an embodiment, the partition-based splits generatorcan calculate the number of bytes, data included for each validpartition, combine or split partitions/sub-partitions in the list ofvalid partitions based on a maximum split size. For example, a partitioncan be divided into multiple splits if the size of the partition isgreater than the maximum split size; and multiple partitions can becombined into one split if their total size is less than or equal to themaximum split size.

In accordance with an embodiment, the partition-based splits generatorcan generate query splits based on the partition-based table splitsdescribed above. The approach generates query splits based solely on thesizes of the partitions and by pruning unnecessarypartitions/sub-partitions in the generating process. In accordance withan embodiment, a user needs not know the datasets in the partitionedtable to use this approach.

FIG. 8 is a flowchart that illustrates a method for dynamicallygenerating partition-based splits based on table properties and/or userpreferences for use in a massively parallel or other distributedenvironment, in accordance with an embodiment. With reference now tothat Figure, a method 800 is shown for database split generation in anassociated massively parallel or other distributed database environmentincluding a plurality of databases and a data warehouse layer providingdata summarization and querying of the plurality of databases. At step802 the method obtains from an associated client application, a queryfor data in a table of the data warehouse layer. The query comprisesquery data representative of a user query and user preference datarepresentative of a user preference.

At step 804 the method obtains, from the data warehouse layer, tabledata representative of one or more properties of the table, wherein thetable data comprises partition data representative of a partition schemeof the table as having a partitioned topology wherein the table islogically divided into one or more partitions or an un-partitionedtopology wherein the table is logically undivided.

At step 806 the method selects a partition-based splits generator inaccordance with one or more of the query data indicating a userpreference for a partition-based splits generator or the partition dataindicating the partition scheme of the table as having the partitionedtopology.

At step 808 the method generates, by the selected partition-based splitsgenerator, one or more table splits for each of the one or morepartitions of the table, the one or more table splits dividing the userquery into a plurality of query splits.

At step 810 the method outputs the plurality of query splits to aplurality of associated mappers for execution by the plurality ofassociated mappers of each of the plurality of query tasks against thetable.

In one embodiment, the generating the one or more table splits for eachof the one or more partitions of the table includes sub-steps ofobtaining partition size data representative of a partition size of eachof the one or more partitions of the table, obtaining split size datarepresentative of a maximum split size of each of the plurality of querysplits, comparing the split size data with the partition size data andstoring a result of the comparing as comparison data, and generating theone or more table splits for each of the one or more partitions of thetable in accordance with the comparison data.

In another embodiment, the generating the one or more table splits foreach of the one or more partitions of the table includes sub-steps ofdetermining the partition size is less than the maximum split size inaccordance with the comparison data, and generating a single table splitfor a plurality of partitions in accordance with determining thepartition size is less than the maximum split size.

Size-Based Splits

In accordance with a further embodiment, a method for size-baseddatabase split generation in an associated massively parallel or otherdistributed database environment including a plurality of databases anda data warehouse layer providing data summarization and querying of theplurality of databases is provided. A query for data in a table of thedata warehouse layer is obtained from an associated client application,the query comprising one or more of query data representative of a userquery that selects all columns in the Hive table or user preference datarepresentative of a user preference. Table data representative of one ormore properties of the table is obtained from the data warehouse layer,and data representative of a data query result size limit is obtainedfrom one or more of the query data or the table data size. The methodfurther includes determining one or more ranges of the table inaccordance with the size data, and selecting a size-based splitsgenerator in accordance with one or more of the query data indicating auser preference for a size-based splits generator or the table dataindicating a size of the table as having predetermined selected size.The method further includes generating, by the selected size-basedsplits generator, a single size-based split query for each of the one ormore ranges of the table, and outputting the single size-based splitquery for each of the one or more ranges of the table to a plurality ofassociated mappers for execution by the plurality of associated mappersof the single size-based split query for each of the one or more rangesagainst the table.

With reference back again to FIG. 6, when it is determined at controlblock 622 that the data in the table is unpartitioned, the databasetable accessor 110 initiates a control function at control block 630 todetermine whether the data in the table is of a fixed size, and acontrol function at control block 640 when the table is not of a fixedsize to determine a number of records in the table. When it isdetermined at control blocks 622, 632 that the data in the table isunpartitioned and not of a fixed size, the split generator selectionlogic 129 selects a limit-based splits protocol at control block 656 forgenerating a limit-based splits generator 121 (FIG. 1). However, when itis determined at control blocks 622, 630 that the data in the table isunpartitioned and of a fixed size, the database table accessor 110 candetermine from the metadata store 150 whether the data in the table isof an unknown database chunk size at control block 640 or whether thedata in the table is of a known size at control block 642. When it isdetermined at control block 640 that the data in the table is of anunknown database chunk size, the split generator selection logic 129selects a size-based splits protocol at control block 652 for generatinga size-based splits generator 121 (FIG. 1).

In accordance with an embodiment and with reference next to FIG. 9, dueto memory constraints, a mapper can only process a certain number oftable records. Feeding too many records into a mapper can cause themapper to hang. As such, it is beneficial to control the size of eachinput split using the size-based split generator at control block 652(FIG. 6).

FIG. 9 illustrates a system for generating size-based splits for use ina massive parallel environment, in accordance with an embodiment, andincludes a database table accessor and a data warehouse layer, where thedatabase table accessor, upon receiving a query for data in a table, canselect a sized-based splits generator based on properties of the tableand user preferences specified in an external table created in the datawarehouse layer. In accordance with an embodiment, the sized-basedsplits generator divides the table into multiple chunks of auser-defined size but the last chunk, creates row ranges for each chunk,and uses the row ranges to split the query into multiple query splits.To generate input splits based on a user-defined size, a user can createan external table in the data warehouse layer with a user-defined sizeas the splitter kind.

As shown in FIG. 9, to generate input splits based on a user-definedsize, a user can create an external table 946 in the data warehouselayer with a user-defined size as the splitter kind, which is furtherillustrated in the example pseudocode of Listing 6 set out below:

Listing 6 CREATE EXTERNAL TABLE sales_test ( prod_id int, cust_nameSTRING, TIME_ID timestamp ) STORED BY‘oracle.hcat.osh.OracleStorageHandler’ TBLPROPERTIES ( ‘oracle.jdbc.url’= ‘jdbc:oracle:thin:@localhost:1521:orcl’, ‘oracle.jdbc.username’ =‘scott’, ‘oracle.jdbc.password’ = ‘tiger’, ‘oracle.hcat.osh.tableName’ =‘sales_test’, ‘oracle.hcat.osh.splitterKind’ = ‘SIZED_RANGE’,‘oracle.hcat.osh.splitterKind’ = ‘256M’ );

As shown in the Listing 6 above, the splitter kind is defined asSIZED-RANGE. In accordance with an embodiment, the‘oracle.hcat.osh.splitterKind’=‘256M’ is not required; the size can bederived from the Hadoop input split size(mapreduce.input.fileinputformat.split.maxsize).

As further shown in FIG. 9, the database table accessor can receive aquery for data in the database table, wherein the query, in accordancewith an embodiment, can select all rows in the table. The database tableaccessor can choose a sized-based splits generator 921 based on thesplitter kind (i.e. user hint/preference including a given split size)and properties of the table, and use the sized-based splits generator tocalculate the total size of the table and divide the table into multiplechunks of the given size.

In accordance with an embodiment, the database table accessor can minethe data dictionary of the table and create row ranges for each chunk ofthe table. In accordance with an embodiment, each row range can have nrows, wherein the sum of the n rows is the user-defined size or adefault size of 128M, but for the last chunk, which can be less than theuser-defined size or the default size.

In accordance with an embodiment, the database table accessor cancalculate the row ranges using the total number of blocks occupied bythe table, the size of each block, and the total number of rows in thetable. For example, using the total number of blocks and the size ofeach block, the database table accessor can figure out the number ofrows per block, and row ranges per block.

In accordance with an embodiment, the database table accessor can usethe row ranges to split the query into multiple query splits, where eachquery split is then embedded into an input split object and provided toa record reader for execution against the external table.

FIG. 10 is a flowchart that illustrates a method 1000 for dynamicallygenerating size-based splits based on table properties and/or userpreferences for use in a massively parallel or other distributedenvironment, in accordance with an embodiment. At step 1002 the methodobtains, from an associated client application, a query for data in atable of the data warehouse layer, the query comprising one or more ofquery data representative of a user query that selects all columns inthe table or user preference data representative of a user preference.

At step 1004 the method obtains, from the data warehouse layer, tabledata representative of one or more properties of the table, and at step1006 the method obtains from one or more of the query data or the tabledata size data representative of a data query result size limit.

At step 1008 the method determines one or more ranges of the table inaccordance with the size data, and at step 1010 the method selects asize-based splits generator in accordance with one or more of the querydata indicating a user preference for a size-based splits generator orthe table data indicating a size of the table as having predeterminedselected size.

At step 1012 the method generates, by the selected size-based splitsgenerator, a single size-based split query for each of the one or moreranges of the table.

The method then outputs the single size-based split query for each ofthe one or more ranges of the table to a plurality of associated mappersfor execution by the plurality of associated mappers of the singlesize-based split query for each of the one or more ranges against thetable.

ROWID Range-Based Splits

In accordance with a further embodiment, a method for ROWID-baseddatabase split generation in an associated massively parallel or otherdistributed database environment including a plurality of databases anda data warehouse layer providing data summarization and querying of theplurality of databases is provided. A query for data in a table of thedata warehouse layer is obtained from an associated client application,the query comprising one or more of query data representative of a userquery that selects all columns in the table or user preference datarepresentative of a user preference. Table data representative of one ormore properties of the table is obtained from the data warehouse layer,the table data comprising files, data blocks, and row numbers for thetable in accordance with the user query for blocks of data in the table.The method further includes creating multiple block ranges for theblocks of data in the table in accordance with the user query, andcreating for each of the multiple block ranges ROWID range data, theROWID range data being representative of pseudo columns of the tablethat serve as unique identifiers for a row of the table. The methodfurther includes selecting a ROWID-based splits generator in accordancewith one or more of the query data indicating the user query thatselects all columns in the table or the user preference data indicatinga user preference for a ROWID-based splits generator, and generating, bythe selected ROWID-based splits generator, table splits dividing theuser query into a plurality of ROWID query splits wherein each of theROWID query splits is associated with one of the multiple block rangesin accordance with the ROWID range data. The method further includesoutputting the plurality of ROWID query splits to a plurality ofassociated mappers for execution by the plurality of associated mappersof the plurality of ROWID query splits against the table.

With reference back again to FIG. 6, when it is determined at controlblock 622 that the data in the table is unpartitioned, the databasetable accessor 110 initiates a control function at control block 630 todetermine whether the data in the table is of a fixed size, and acontrol function at control block 640 when the table is not of a fixedsize to determine a number of records in the table. When it isdetermined at control blocks 622, 632 that the data in the table isunpartitioned and not of a fixed size, the split generator selectionlogic 129 selects a limit-based splits protocol at control block 656 forgenerating a limit-based splits generator 121 (FIG. 1). However, when itis determined at control blocks 622, 630 that the data in the table isunpartitioned and of a fixed size, the database table accessor 110 candetermine from the metadata store 150 whether the data in the table isof an unknown database chunk size at control block 640 or whether thedata in the table is of a known size at control block 642. When it isdetermined at control block 642 that the data in the table is of anknown database chunk size, the split generator selection logic 129selects a ROWID-based splits protocol at control block 654 forgenerating a ROWID-based splits generator 121 (FIG. 1).

As noted above, when it is determined at control block 642 that the datain the table is of an known database chunk size, the split generatorselection logic 129 selects a ROWID-based splits protocol at controlblock 654 for generating a ROWID-based splits generator 121 (FIG. 1). Inthis regard and in accordance with an embodiment, every record in adatabase table can be associated with a ROWID, a pseudo column thatuniquely identifies a row within the table. In accordance with anembodiment, each ROWID can change when a table is reorganized, exportedor imported. In accordance with an embodiment, in a partitioned table, aROWID can also change when a row corresponding to the ROWID migratesfrom one partition to another.

In accordance with an embodiment, ROWIDs need not be physically storedin a database. For example, an Oracle database can generate a ROWID whenthe ROWID is queried.

In accordance with an embodiment, a ROWID can be associated with a filenumber, a block number, and a row number in which a row is stored.

FIG. 11 illustrates a system for generating ROWID-based splits for usein a massive parallel environment, in accordance with an embodiment. Asshown in FIG. 11, in accordance with an embodiment, an external table1146 can be created for a database table, wherein the external table ismanaged by a database table accessor and with a ROWID range specified asthe splitter kind (user hint/preference).

In accordance with an embodiment, an example DDL syntax for creatingsuch an external table in the data warehouse layer can be illustrated inListing 7, where a splitter kind is defined to be ROWID_RANGE.

Listing 7  CREATE EXTERNAL TABLE sales_test ( prod_id int, cust_nameSTRING, TIME_ID timestamp ) STORED BY‘oracle.hcat.osh.OracleStorageHandler’ TBLPROPERTIES ( ‘oracle.jdbc.url’= ‘jdbc:oracle:thin:@localhost:1521:orcl’, ‘oracle.jdbc.username’ =‘scott’, ‘oracle.jdbc.password’ = ‘tiger’, ‘oracle.hcat.osh.tableName’ =‘sales_test’, ‘oracle.hcat.osh.splitterKind’ = ‘ROWID_RANGE’; );

As shown in Listing 7, the value of the ROWID_RANGE is not specified. Inaccordance with an embodiment, when a value of the splitter kind is notspecified, a default value can be used.

As further shown in FIG. 11, in accordance with an embodiment, when auser or an application issues a query that selects all rows in thedatabase table, the database table accessor, based on properties of thedatabase table and the user hint obtained 1151 from the data warehouselayer, can choose a ROWID-based splits generator 1121. In accordancewith an embodiment, the database table accessor can then obtain files,data blocks and row numbers for the table, and create multiple blockranges from the blocks selected, where each block range has a fixed-sizenumber of blocks but for the last range.

In accordance with an embodiment, when a user or an application issues aquery that selects all rows in the database table, the database tableaccessor, based on properties of the database table and the user hintobtained from the data warehouse layer, can choose a ROWID-based splitsgenerator. In accordance with an embodiment, the database table accessorcan then obtain files, data blocks and row numbers for the table, andcreate multiple block ranges from the blocks selected, where each blockrange has a fixed-size number of blocks but for the last range.

In accordance with an embodiment, the database table accessor can createROWIDs for each block range using files and row numbers, wherein oneROWID is created per block, which makes one ROWID range per block range.

In accordance with an embodiment, the database table accessor can splitthe query into multiple smaller or query splits based on the ROWIDranges created on each block range, where each query split is thenembedded into an input split object and provided to a record reader forexecution against the external table.

In accordance with an embodiment, the ROWID range splitter kinds can bedynamically modified during implementation. For example, for thepartition-based splitter kind, if a large partition is detected thatdoes not contain much data, the database table accessor can merge thepartition with one or more other partitions. Similarly, if a partitionis detected that contains too much data, the partition can be split intomultiple smaller partitions.

FIG. 12 is a flowchart that illustrates a method for dynamicallygenerating ROWID-based splits based on table properties and/or userpreferences for use in a massively parallel or other distributedenvironment, in accordance with an embodiment. With reference now tothat Figure, a method 1200 is illustrated for database ROWID-based splitgeneration in an associated massively parallel or other distributeddatabase environment including a plurality of databases and a datawarehouse layer providing data summarization and querying of theplurality of databases. The method 1200 obtain at step 1202, from anassociated client application, a query for data in a table of the datawarehouse layer, the query comprising one or more of query datarepresentative of a user query that selects all columns in the table oruser preference data representative of a user preference.

At step 1204 the method obtains, from the data warehouse layer, tabledata representative of one or more properties of the table, the tabledata comprising files, data blocks, and row numbers for the table inaccordance with the user query for blocks of data in the table.

At step 1206 the method creates multiple block ranges for the blocks ofdata in the table in accordance with the user query, and at step 1208the method creates for each of the multiple block ranges ROWID rangedata, the ROWID range data being representative of pseudo columns of thetable that serve as unique identifiers for a row of the table.

At step 1210 the method selects a ROWID-based splits generator inaccordance with one or more of the query data indicating the user querythat selects all columns in the table or the user preference dataindicating a user preference for a ROWID-based splits generator.

At step 1212 the method generates, by the selected ROWID-based splitsgenerator, table splits dividing the user query into a plurality ofROWID query splits wherein each of the ROWID query splits is associatedwith one of the multiple block ranges in accordance with the ROWID rangedata.

At step 1214 the method outputs the plurality of ROWID query splits to aplurality of associated mappers for execution by the plurality ofassociated mappers of the plurality of ROWID query splits against thetable.

High-Speed Data Transfer from JDBC

FIG. 13 illustrates a system for providing rapid transfer of data from adatabase driver to generic writable interfaces in a data warehouselayer, in accordance with an embodiment. in particular, in accordancewith an embodiment, the figure illustrates a system and method forhigh-speed data transfer from JDBC to a data warehouse for use with amassively parallel or other distributed database environment forproviding high speed data transfer between a plurality of databases ofthe database environment and a data warehouse layer providing datasummarization and querying of the plurality of databases. The method ofthe illustrated embodiment includes obtaining, from an associated clientapplication, a query for data in a table of the data warehouse layer,the query comprising query data representative of a user query and userpreference data representative of a user preference. The method furtherincludes obtaining table data representative of one or more propertiesof the table, determining a splits generator in accordance with one ormore of the user preference or the one or more properties of the table,and generating, by the selected splits generator, table splits dividingthe user query into a plurality of query splits. The method furtherincludes outputting the plurality of query splits to a database drivercomprising a data transformation logic component and a fast pathapplication programming interface (API), and receiving result dataderived from executing the plurality of queries by associated mappersagainst the table. The method further includes selectively transforming,by fast path API of the database driver, the result data from a formatof the table to a data mining tool format directly usable by one or morecomponents of the associated data warehouse layer.

As described above, the database table accessor can be used to retrievedata from a relational database via a data warehouse layer (e.g., HIVE)and transfer the data to a data warehouse layer (e.g., HIVE) for datamining.

The database table accessor typically obtains the data using methodssuch as getDate( ), getTimestamp( ) and getString( ) on a ResultSet in adatabase driver, e.g., a JDBC driver. These methods convert the datafrom database formats to Java core formats such as java.sql.Date,java.sql.Timestamp, java.lang.String. In accordance with an embodiment,Java objects can be immutable objects that cannot be changed oncecreated. As such, they can be multiple copies of a data object, whichstress the heap and impact performance for data mining.

Data mining tools such as HIVE needs a high speed data transfermechanism without multiple conversions from e.g., Oracle databaseformats to Java formats and then to data mining tool formats; andwithout multiple copies of a data object.

In accordance with an embodiment, described herein is a system andmethod for providing rapid transfer of data from a database driver togeneric writable interfaces in a data warehouse layer (e.g., HIVE).

In accordance with an embodiment, the system includes a database driverused to obtain data from a database table to a data warehouse layer(e.g., HIVE), where the database driver includes a data transformationlogic component and a fast path API. When a result set is returned fromthe database table, the database driver can perform a check to determinewhether a fast conversion capability is supported, and if it is,transform the data from database formats to data mining tool formatsusing the fast path API and in accordance with a logic defined in thedata transformation logic component.

As shown in FIG. 13, a database driver 1363, e.g, a JDBC driver, can beused to retrieve data from a database table 105 for use by the datawarehouse layer. The database driver includes a fast path API 1365 and adata transformation logic component 1367. When data is returned from thedatabase table via the external table, the database driver candynamically check the capability of fast-conversion in the databasedriver and cache the state. In accordance with an embodiment, thecapability check can be performed using Java reflection to examinemethods in the database driver.

In accordance with an embodiment, if a fast conversion capability issupported by the database driver, the fast path API can be used toperform the data transformation in accordance with a logic defined inthe data transformation logic component. In accordance with anembodiment, the data transformation logic can be defined by users basedon their data mining needs.

In accordance with an embodiment, data types in data mining tool formatscan be similar to Java object types, e.g., a string object. However, newobjects need not be created every time; instead, a same object can beused for holding intermediate data for data mining. As such, multiplecopies of a same data are can be avoided.

The following pseudo code illustrates an example implementation of themethod for providing rapid transfer of data from a database driver togeneric writable interfaces in a data warehouse layer, in accordancewith an embodiment.

Listing 8 oracle.jdbc.internal.OracleResultSet: /** * @returns thenumber of bytes copied into the buffer */ public int getBytes(intparameterIndex, byte[ ] buffer, int offset) throws SQLException;DynamicByteArray::getBytes int getBytes(int dataOffset, int dataLength,byte[ ] buffer, int byteOffset): if(data == CHAR OR data == VARCHAR)if(uniByte OR networkCharSet ==UTF8) Then copy-bytes-to-buffer; elseThen convert to UTF8 bytes; DateAccessor::getBytes int getBytes(introwIndex, byte[ ] buffer, int byteOffset): if(isNull( )) return 0 elseconvert-date-to-seconds since 1 Jan 1970 Serialize seconds onto byte[ ]return 4 // size of int TimestampAccessor::getBytes int getBytes(introwIndex, byte[ ] buffer, int byteOffset): if(isNull( )) return 0 elseconvert-date-to-milli-seconds since 1 Jan 1970 Serialize seconds ontobyte[ ] Serialize nano-seconds to byte[ ] return 12 // size of long + 4bytes nano seconds Reader:refreshColumns: Dynamically check thecapability of fast-conversion in the driver Cache the state if(fastPath)use fast path API to get byte[ ] use byte[ ] with Writable else useregular access

FIG. 14 is a flowchart that illustrates a method 1400 for rapidtransmission of data such as Oracle Java Database Connectivity (JDBC)data to a writable format such as a HADOOP writable format. Inaccordance with an embodiment and with reference now to that Figure, amethod 1400 is illustrated for use with a massively parallel or otherdistributed database environment for providing high speed data transferbetween a plurality of databases of the database environment and a datawarehouse layer providing data summarization and querying of theplurality of databases. At step 1402 the method obtains, from anassociated client application, a query for data in a table of the datawarehouse layer, the query comprising query data representative of auser query and user preference data representative of a user preference.

At step 1404 the method obtains table data representative of one or moreproperties of the table, and at step 1406 the method determines a splitsgenerator in accordance with one or more of the user preference or theone or more properties of the table.

At step 1408 the method generates, by the selected splits generator,table splits dividing the user query into a plurality of query splits,and at step 1410 outputs the plurality of query splits to a databasedriver comprising a data transformation logic component and a fast pathapplication programming interface (API).

At step 1412 the method receives result data derived from executing theplurality of queries by associated mappers against the table, and atstep 1414 the method selectively transforms, by fast path API of thedatabase driver, the result data from a format of the table to a datamining tool format directly usable by one or more components of theassociated data warehouse layer.

Optimized Query Processing with Table-Level Predicate Pushdown

When a query is executed in a data warehouse layer (e.g., HIVE), thedata is selected and query predicates are applied at the data warehouselayer to filter the rows that the query does not cover. A query executedas described above can have poor performance since the whole table istransported to a data warehouse layer for row trimming. In addition,such queries may not even be executed because of memory constraints ifthe table is large.

In accordance with an embodiment, predicate pushdown is a feature wherequery predicates are pushed down to the storage nodes for analysis forhigh throughput. In a query, e.g., a SQL statement, portions of the SQLstatement, specifically ones that filter data, are referred to aspredicates. By pushing some query predicates down to a level as low aspossible, query performance can be improved by avoiding transferring allthe rows in a table to a data warehouse layer when only one or two rowsthat a query is to retrieve.

In accordance with an embodiment, described herein is a system andmethod for optimizing query processing with predicates pushdown from adata warehouse layer to table level in a database. In accordance with anembodiment, predicate pushdown at the table level can be achieved byusing standard properties of an external table, wherein the externaltable is created with a query and predicates of the query for pushdowndefined as table properties. When a query for selecting rows in thetable is received, the database table accessor operates to choose anappropriate splits generator, use a query formulator to formulate thequery so that the query predicates can be executed at the table level.

In accordance with an example embodiment, a method is provided for splitquery processing with table level predicate pushdown in a massivelyparallel or other distributed database environment including a pluralityof databases and a data warehouse layer providing data summarization andquerying of the plurality of databases. The method includes obtaining,from an associated client application, a query for data in a table ofthe data warehouse layer, the query comprising query data representativeof a user query, query predicate data representative of one or morepredicates of the user query, and user preference data representative ofa user preference, wherein the one or more predicates of the user querycomprise portions of the user query operative to filter data. The methodfurther includes obtaining table data representative of one or moreproperties of the table, determining a splits generator in accordancewith one or more of the user preference or the one or more properties ofthe table, and generating, by the selected splits generator, tablesplits dividing the user query into a plurality of query splits. Themethod further includes formulating the plurality of query splits toinclude the query predicate data representative of the one or morepredicates of the user query, and outputting the plurality of querysplits to an associated plurality of mappers for execution by theassociated plurality of mappers of each of the plurality of query splitsagainst the table, wherein the one or more predicates of the user querymay be executed at a logical execution level of the table.

FIG. 15 illustrates a system for optimizing query processing withpredicates pushdown to table level in a database from a data miningtool, in accordance with an embodiment.

As shown in FIG. 15, an external table 1546 can be created with a queryand query predicates for pushdown to the table level defined 1544 asproperties of the external table wherein the external table is managedby the database table accessor and can be used to access the databasetable.

In accordance with an embodiment, an example DDL syntax for creatingsuch an external table in the data warehouse layer can be illustrated inListing 9.

Listing 9 CREATE EXTERNAL TABLE sales_q2 ( prod_id int, cust_nameSTRING, TIME_ID timestamp ) STORED BY‘oracle.hcat.osh.OracleStorageHandler’ TBLPROPERTIES (‘mapreduce.jdbc.url’ = ‘jdbc:oracle:thin:@adc2171607:9221:dbjMt’,‘mapreduce.jdbc.username’ = ‘scott’, ‘mapreduce.jdbc.password’ =‘tiger’, ‘mapreduce.jdbc.input.table.name’ = ‘sales_test’,‘oracle.hcat.osh.splitterKind’ = ‘PARTITIONED_TABLE’,‘mapreduce.jdbc.input.query’ = ‘SELECT prod_id, cust_name, time_id fromsales_test’, ‘mapreduce.jdbc.input.conditions’ = ‘time_id =to_date(\‘10-jun-2006\’,\‘dd-mon-yyyy\’)’ );

As illustrated in Listing 9, an external table is created with a queryand query predicates for pushdown to the table level specified in theexternal table definition.

As further shown in FIG. 15, in accordance with an embodiment, when aquery is received that selects all rows in the database table, thedatabase table accessor can choose an appropriate splits generator 1521to generate query splits, and additionally identify if the query isdefined as parameters or properties of the external table usinginformation retrieved 1551 from the data warehouse layer. The databasetable accessor subsequently can invoke a query formulator 1524 toformulate each query split based on the information from the externaltable metadata store in a way that the predicates defined for pushdowntherein are added to a SQL statements and passed to the database forexecution at the table level.

FIG. 16 is a flowchart that illustrates a method 1600 for providingquery processing with table-level predicate pushdown in a data warehouselayer, in accordance with an embodiment and with reference now to thatFigure, a method 1600 is illustrated for split query processing withtable level predicate pushdown in a massively parallel or otherdistributed database environment including a plurality of databases anda data warehouse layer providing data summarization and querying of theplurality of databases. The method at step 1602 obtains, from anassociated client application, a query for data in a table of the datawarehouse layer, the query comprising query data representative of auser query, query predicate data representative of one or morepredicates of the user query, and user preference data representative ofa user preference, wherein the one or more predicates of the user querycomprise portions of the user query operative to filter data.

At step 1604 the method obtains table data representative of one or moreproperties of the table, at step 1606 determines a splits generator inaccordance with one or more of the user preference or the one or moreproperties of the table, and at step 1608 generates, by the selectedsplits generator, table splits dividing the user query into a pluralityof query splits.

At step 1610 the method formulates the plurality of query splits toinclude the query predicate data representative of the one or morepredicates of the user query.

At step 1612 the method outputs the plurality of query splits to anassociated plurality of mappers for execution by the associatedplurality of mappers of each of the plurality of query splits againstthe table, wherein the one or more predicates of the user query may beexecuted at a logical execution level of the table.

Efficient Connection Management

A data processing task, e.g, a Hadoop task, is a unit of work submittedby a user to a data processing cluster, e.g., Hadoop, for processing. Inaccordance with an embodiment, query splits from a query are notexecuted atomically; instead, they are executed at different points intime. Each query split is equipped with a system change number (SCN)that make the reads from each query split consistent.

Each task processes a split of a large dataset generated by an inputformat component and each task uses a database connection forinteracting with a database. Every time a new task is submitted forprocessing, a separate process, such as Java virtual machine process,can be created and a database connection is created. In a cluster withthousands of nodes, each of which can process hundreds of tasks, thenumber of database connections used can create a connection storm on thedatabase.

In accordance with embodiments herein a system and method is providedfor efficient connection management in a massively parallel or otherdistributed database environment including a plurality of databases anda data warehouse layer providing data summarization and querying of theplurality of databases. In accordance with an example embodiment, themethod includes obtaining, from an associated client application, aquery for data in a table of the data warehouse layer, the querycomprising query data representative of a user query and user preferencedata representative of a user preference. The method further includesrequesting a connection from a connection pool to obtain via theconnection table data representative of one or more properties of thetable, determining a splits generator in accordance with one or more ofthe user preference or the one or more properties of the table, andgenerating, by the selected splits generator, table splits dividing theuser query into a plurality of query splits. The method further includesoutputting the plurality of query splits to an associated plurality ofmappers for execution using the connection by the associated pluralityof mappers of each of the plurality of query splits against the table.

In accordance with an embodiment, described herein is a system andmethod for efficient connection management in a distributed parallelprocessing environment. In accordance with an embodiment, a pseudoconnection pool of size one is created. The connection pool creates aconnection when a connection is first requested by the database tableaccessor to create query splits, wherein the connection is used byvarious components, e.g., a record reader and a mapper, for processing afirst split. When a connection is requested by a task that reuses theprocess that created for processing the first split, the same connectionis returned for use from the connection pool when the connectioncredentials match. If the connection credentials do not match, theexisting connection is closed, and a new connection is created andreturned.

In accordance with an embodiment, a task reuse parameter can be used toenable a process to process a plurality of tasks, so that the pluralityof tasks can reuse a same connection from the connection pool.

In accordance with an embodiment, the connection pool can cache a SQLstatement so that query splits from a same query can reuse the SQLstatement. In accordance with an embodiment, query splits from a samequery include a same SQL statement with different binds. Caching the SQLstatement can improve performance. For example, by caching executablestatements used repeatedly, performance be can improved by preventingthe overhead of repeated cursor creation, repeated statement parsing andcreating, and by reusing data structures in the data warehouse layer.

FIG. 17 illustrates a system for efficient connection management in adistributed parallel processing environment, in accordance with anembodiment. As illustrated, the database table accessor includes aconnection pool 1701 of size one and caches a single connection. Theconnection can be used to connect 1707 to a database table via theexternal table.

In accordance with an embodiment, when a query is received, the databasetable accessor can request a connection from the connection pool forretrieving user hints and table properties to generate query splits. Theconnection pool can create a connection 903 and returns the connectionto the database table accessor. When a query split, e.g., Input Split A128, is executed by a record reader, e.g., 152, against the externaltable, the same connection is used to retrieve table records forprocessing by the data processing cluster.

As further shown, the data processing cluster includes two processes(e.g., Java virtual machine processes) 1772 and 1777, and each processincludes a process reuse parameter 1774 and 1778. In accordance with anembodiment, a process reuse parameter can be set by a processing taskthat spawns the process. If the value is 1 (the default), then a processwith the value cannot be reused. A higher value than 1 can be specifiedvia an application programming interface to enable the process toprocess more tasks. If the value is −1, there is no limit to the numberof tasks the processes can run (of the same job). In accordance with anembodiment, the process reuse parameter, e.g., reuse.jvm.num.tasks, canbe associated with the task that spawns the process, which canautomatically inherit the value of the parameter.

As illustrated by FIG. 17, Tasks A 1775 and B 1776 use the same process.In accordance with an embodiment, which tasks share a certain processcan be determined by the data processing cluster. In accordance with anembodiment, Task B, which is spawn to process input split B 130, canrequest a connection from the connection pool. If the connectioncredentials match, the existing connection is returned for use by therecord reader B 154 and the processing task B. As such, the connectionin the connection pool can be shared between tasks and among variouscomponents within a task. In accordance with an embodiment, if theconnection credentials do not match, the existing connection is closedand a new connection 905 can be created and returned.

In accordance with an embodiment, the connection pool of size one canprovide a connection to be reused by a next task that reuses the sameprocess in the data processing cluster. In addition, this approach forsharing database resources across multiple tasks in a distributed systemcan provide additional benefits such as performance enhancingpersistence between tasks, when compared with alternative ways tocontrol the number of connections, e.g., by using a proxy or a walletmanager.

FIG. 18 is a flowchart that illustrates a method 1800 for providingefficient connection management in a distributed database environment,in accordance with an embodiment and, with reference now to that Figure,a method 1800 is provided for efficient connection management in amassively parallel or other distributed database environment including aplurality of databases and a data warehouse layer providing datasummarization and querying of the plurality of databases. The method atstep 1802 obtains, from an associated client application, a query fordata in a table of the data warehouse layer, the query comprising querydata representative of a user query and user preference datarepresentative of a user preference.

At step 1804 the method requests a connection from a connection pool toobtain via the connection table data representative of one or moreproperties of the table, and at step 1806 determines a splits generatorin accordance with one or more of the user preference or the one or moreproperties of the table.

At step 1808 the method generates, by the selected splits generator,table splits dividing the user query into a plurality of query splits.

At step 1810 the method outputs the plurality of query splits to anassociated plurality of mappers for execution using the connection bythe associated plurality of mappers of each of the plurality of querysplits against the table.

In accordance with a further embodiment, the method is operative at step1812, to cache a structured query language (SQL) statement associatedwith the user query in the connection pool whereby each of the pluralityof query splits generated by splits dividing the user query into theplurality of query splits can selectively reuse the connection and theSQL statement.

Embodiments herein may be conveniently implemented using one or moreconventional general purpose or specialized digital computer, computingdevice, machine, or microprocessor, including one or more processors,memory and/or computer readable storage media programmed according tothe teachings of the present disclosure. Appropriate software coding canreadily be prepared by skilled programmers based on the teachings of thepresent disclosure, as will be apparent to those skilled in the softwareart.

In some embodiments, the embodiment includes a computer program productwhich is a non-transitory storage medium or computer readable medium(media) having instructions stored thereon/in which can be used toprogram a computer to perform any of the processes of the presentinvention. Examples of the storage medium can include, but is notlimited to, any type of disk including floppy disks, optical discs, DVD,CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs,EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards,nanosystems (including molecular memory ICs), or any type of media ordevice suitable for storing instructions and/or data.

The foregoing description of embodiments have been provided for thepurposes of illustration and description. It is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Many modifications and variations will be apparent to the practitionerskilled in the art. The embodiments were chosen and described in orderto best explain the principles of the invention and its practicalapplication, thereby enabling others skilled in the art to understandthe invention for various embodiments and with various modificationsthat are suited to the particular use contemplated. It is intended thatthe scope of the invention be defined by the following claims and theirequivalents.

What is claimed is:
 1. A method for use with a massively parallel ordistributed database environment for providing high speed data transferbetween a plurality of databases of the database environment and a datawarehouse layer providing data summarization and querying of theplurality of databases, the method comprising: obtaining, from anassociated client application, a query for data in a table of the datawarehouse layer, the query comprising query data representative of auser query and user preference data representative of a user preference;obtaining table data representative of one or more properties of thetable; determining a splits generator in accordance with one or more ofthe user preference or the one or more properties of the table;generating, by the selected splits generator, table splits dividing theuser query into a plurality of query splits; and outputting theplurality of query splits to a database driver comprising a datatransformation logic component and a fast path application programminginterface (API); receiving result data derived from executing theplurality of queries by associated mappers against the table;selectively transforming, by fast path API of the database driver, theresult data from a format of the table to a data mining tool formatdirectly usable by one or more components of the associated datawarehouse layer.
 2. The method of claim 1, wherein the selectivelytransforming the result data comprises performing a capability checkusing Java reflection to examine methods in the database driver.
 3. Themethod of claim 1, wherein the determining the splits generatorcomprises selecting the splits generator from among an enumeration ofsplitter kinds in accordance with the user preference and the one ormore properties of the table.
 4. The method of claim 1, wherein thedetermining the splits generator comprises determining the splitsgenerator in accordance exclusively with the one or more properties ofthe table.
 5. The method of claim 1, wherein the determining the splitsgenerator comprises determining the splits generator in accordanceexclusively with the user preference.
 6. The method of claim 1, whereinthe obtaining the table data comprises obtaining metadata representativeof the one or more properties of the table.
 7. The method of claim 1,wherein: the obtaining the table data comprises obtaining table datarepresentative of one or more properties of an external table in a HIVEdata warehouse layer; and the outputting the plurality of query splitscomprises outputting the plurality of query splits to the associatedplurality of mappers for execution by the associated plurality ofmappers as HADOOP tasks.
 8. A system for use with a massively parallelor distributed database environment for providing high speed datatransfer between a plurality of databases of the database environmentand a data warehouse layer providing data summarization and querying ofthe plurality of databases, the system comprising: one or moremicroprocessors; a database table accessor running on the one or moremicroprocessors, wherein the database table accessor operates to performsteps comprising: obtaining, from an associated client application, aquery for data in a table of the data warehouse layer, the querycomprising query data representative of a user query and user preferencedata representative of a user preference; obtaining table datarepresentative of one or more properties of the table; determining asplits generator in accordance with one or more of the user preferenceor the one or more properties of the table; generating, by the selectedsplits generator, table splits dividing the user query into a pluralityof query splits; and outputting the plurality of query splits to adatabase driver comprising a data transformation logic component and afast path application programming interface (API); receiving result dataderived from executing the plurality of queries by associated mappersagainst the table; selectively transforming, by fast path API of thedatabase driver, the result data from a format of the table to a datamining tool format directly usable by one or more components of theassociated data warehouse layer.
 9. The system of claim 8, wherein theselectively transforming the result data comprises performing acapability check using Java reflection to examine methods in thedatabase driver.
 10. The system of claim 8, wherein the determining thesplits generator comprises selecting the splits generator from among anenumeration of splitter kinds in accordance with the user preference andthe one or more properties of the table.
 11. The system of claim 8,wherein the determining the splits generator comprises determining thesplits generator in accordance exclusively with the one or moreproperties of the table.
 12. The system of claim 8, wherein thedetermining the splits generator comprises determining the splitsgenerator in accordance exclusively with the user preference.
 13. Thesystem of claim 8, wherein the obtaining the table data comprisesobtaining metadata representative of the one or more properties of thetable.
 14. The system of claim 8, wherein: the obtaining the table datacomprises obtaining table data representative of one or more propertiesof an external table in a HIVE data warehouse layer; and the outputtingthe plurality of query splits comprises outputting the plurality ofquery splits to the associated plurality of mappers for execution by theassociated plurality of mappers as HADOOP tasks.
 15. A non-transitorycomputer readable storage medium, including instructions stored thereonwhich when read and executed by one or more computers of a databasetable accessor in a massively parallel or distributed databaseenvironment including a plurality of databases and a data warehouselayer providing data summarization and querying of the plurality ofdatabases, cause the one or more computers of the database tableaccessor to perform the steps comprising: obtaining, from an associatedclient application, a query for data in a table of the data warehouselayer, the query comprising query data representative of a user queryand user preference data representative of a user preference; obtainingtable data representative of one or more properties of the table;determining a splits generator in accordance with one or more of theuser preference or the one or more properties of the table; generating,by the selected splits generator, table splits dividing the user queryinto a plurality of query splits; and outputting the plurality of querysplits to a database driver comprising a data transformation logiccomponent and a fast path application programming interface (API);receiving result data derived from executing the plurality of queries byassociated mappers against the table; selectively transforming, by fastpath API of the database driver, the result data from a format of thetable to a data mining tool format directly usable by one or morecomponents of the associated data warehouse layer.
 16. Thenon-transitory computer readable storage medium of claim 15, wherein theselectively transforming the result data comprises performing acapability check using Java reflection to examine methods in thedatabase driver.
 17. The non-transitory computer readable storage mediumof claim 15, wherein the determining the splits generator comprisesselecting the splits generator from among an enumeration of splitterkinds in accordance with the user preference and the one or moreproperties of the table.
 18. The non-transitory computer readablestorage medium of claim 15, wherein the determining the splits generatorcomprises determining the splits generator in accordance exclusivelywith the user preference.
 19. The non-transitory computer readablestorage medium of claim 15, wherein the obtaining the table datacomprises obtaining metadata representative of the one or moreproperties of the table.
 20. The non-transitory computer readablestorage medium of claim 15, wherein: the obtaining the table datacomprises obtaining table data representative of one or more propertiesof an external table in a HIVE data warehouse layer; and the outputtingthe plurality of query splits comprises outputting the plurality ofquery splits to the associated plurality of mappers for execution by theassociated plurality of mappers as HADOOP tasks.